System and methods involving edge camera assemblies in handheld devices

ABSTRACT

Systems and methods are disclosed for capturing and displaying real-time images of the environment in front of a user. In one implementation, a casing for a mobile device having an upward facing display surface and an opposing lower surface is provided. The casing may include a housing and image sensor having at least one optical axis configured to be oriented at a fixed obtuse angle with respect to the lower surface when the mobile device is retained by the housing. The image sensor may be oriented such that when the housing is held by a walking user at an acute angle with respect to a plane on which the user walks, the at least one optical axis generally faces in a walking direction of the user. The casing may further include circuitry for conveying from the at least one image sensor, image data for real-time display on the display surface.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 62/333,197, filed on May 7, 2016; U.S. patentapplication Ser. No. 15/220,418, filed on Jul. 27, 2016; U.S.Provisional Patent Application No. 62/425,741, filed on Nov. 23, 2016;and U.S. Provisional Patent Application No. 62/434,695, filed on Dec.15, 2016. All of the foregoing applications are incorporated herein byreference in their entirety.

BACKGROUND I. Technical Field

The present disclosure relates generally to image capturing in handheldmobile communications devices and, more specifically, to systems andmethods for capturing and displaying real-time images of the environmentin front of a user.

II. Background Information

Mobile communications devices are a prevalent part of modern daily life.Although originally conceived as a communications device to providespeech communication between people, mobile communications devices havebecome central to people's interactions, communications, and activitieswith each other and with online data services. It is a common situationthat when a user's attention is directed toward the screen of the mobilecommunications device, the user is disengaged from the surroundingenvironment.

This situation has hazardous implications as many people use theirmobile communications devices while doing other activities, such aswalking on the street. For example, people walking down a street whiletyping a text onto the mobile communications device have been seenwalking into lampposts, puddles, and even oncoming vehicular traffic.Due to the angle at which users usually look at mobile device screens,the users' peripheral vision cannot be relied on to pick out obstaclesin their walking direction.

Moreover, personal mobile communications devices are being used todayfor generating augmented reality. In some cases, apps installed onmobile communications devices may use the devices' GPS and camera tosuperimpose content atop a user's real-time view of the environment.These apps can create a composite presentation rooted in both real andvirtual worlds. One example is the “Pokémon Go” app, which became aglobal phenomenon and was one of the most used and profitable mobileapps in 2016. However, the current placement of cameras in mobilecommunications devices requires holding the mobile communications devicevertically in order to capture the environment in front of the user.This posture is uncomfortable when being used for a long period of timeand not suitable for use while walking.

The disclosed devices and methods are directed to provide a new way forimage capturing, one that aligns with the natural holding angle of themobile communications device and solves at least some of the problemsoutlined above.

SUMMARY

Embodiments consistent with the present disclosure provide devices andmethods for image capturing in handheld mobile communications devices.In one embodiment an image capture casing for a handheld mobilecommunications device having an upward facing display surface and anopposing lower surface is provided. The image capture casing may includea housing for surrounding at least a portion of the handheld mobilecommunications device. The image capture casing may also include atleast one image sensor mounted in the housing and having at least oneoptical axis configured to be oriented at a fixed obtuse angle withrespect to the lower surface when the handheld mobile communicationsdevice is retained by the housing, wherein the at least one image sensoris oriented such that when the housing is held by a walking user at anacute angle with respect to a plane on which the user walks, the atleast one optical axis generally faces in a walking direction of theuser. The image capture casing may further include circuitry forconveying from the at least one image sensor, image data for real-timedisplay on the display surface.

In accordance with another disclosed embodiment, a handheld mobilecommunications device is provided. The handheld mobile communicationsdevice may include, a casing, an upward facing display surface on afirst side of the casing, and a lower surface opposing the displaysurface on a second opposing side of the casing. The handheld mobilecommunications device may also include at least one image sensor mountedin the casing and having at least one optical axis configured to beoriented at a fixed obtuse angle with respect to the lower surface,wherein the at least one image sensor is oriented such that when thecasing is held by a walking user at an acute angle with respect to aplane on which the user walks, the at least one optical axis generallyfaces in a walking direction of the user. The handheld mobilecommunications device may further include circuitry for conveying fromthe at least one image sensor, image data for real-time display on thedisplay surface.

In accordance with another disclosed embodiment, a removable case for ahandheld mobile communications device having an upward facing displaysurface and an opposing lower surface is provided. The case may includea housing for surrounding at least a portion of the handheld mobilecommunications device. The case may also include a connector for makingelectrical connection with a port in the handheld mobile communicationsdevice when the handheld mobile communications device is seated in thehousing. In addition, the case may also at least one image sensormounted in the housing and having at least one optical axis configuredto be oriented at a fixed obtuse angle with respect to the lower surfacewhen the handheld mobile communications device is retained by thehousing, wherein the at least one image sensor is oriented such thatwhen the housing is held by a walking user at an acute angle withrespect to a plane on which the user walks, the at least one opticalaxis generally faces in a walking direction of the user. The case mayfurther include circuitry for conveying from the at least one imagesensor, image data for real-time display on the display surface.

In accordance with another disclosed embodiment, a mobile device forpresenting a real-time view of an environment in front of a user whilethe user interacts with displayed information is provided. The mobiledevice may include a housing generally having a central axial plane, atouch-sensitive display on a first side of the housing, a back on asecond side of the housing, opposite the first side, and a peripheraledge surface between the first side and the second side. The mobiledevice may also include at least one image sensor fixedly secured withinthe housing and having a fixed optical axis at a non-orthogonal anglerelative to each of the first side of the housing, the second side ofthe housing, and the central axial plane.

In accordance with another disclosed embodiment, a removable case for amobile device having a touch-sensitive display is provided. The case mayinclude a housing for surrounding at least a portion of the mobiledevice, the housing having a first side, a second side opposite thefirst side, a peripheral edge surface interconnect the first side andthe second side and having a generally central axial plane. The case mayalso include a connector for making electrical contact with a port inthe mobile device when the mobile device is seated in the housing. Thecase may also include at least one image sensor fixedly secured withinthe housing and having a fixed optical axis at a non-orthogonal anglerelative to each of the first side of the housing, the second side ofthe housing, and the central axial plane. The case may further includecircuitry for conveying from the at least one image sensor, image datafor real-time display on the touch-sensitive display surface.

In accordance with another disclosed embodiment, a handheldcommunications device configured to provide augmented reality to a userlooking downward while traversing a forward path of travel is provided.The handheld communications device may include a housing having a uppersurface, a lower surface, and a side edge, a touch-sensitive displaylocated in the upper surface, and at least one processor. The handheldcommunications device may also include an image sensor fixedly embeddedin the housing at an orientation such that when the handheldcommunications device is held in a hand of a user with thetouch-sensitive display opposed to a face of the downward looking user,an optical axis of the image sensor extends generally in a direction ofthe path of travel. The at least one processor may be programmed toacquire an image stream from the image sensor; identify in the acquiredimage stream an object in the path of travel of the user; look-upinformation about the identified object; present, on the touch-sensitivedisplay in real-time the acquired image stream including the identifiedobject; and present on the touch-sensitive display a visual indicatorassociated with the identified object.

In accordance with another disclosed embodiment, a removable case for ahandheld communications device including a touch-sensitive display isprovided. The case may include a housing having a lower surface, anupper surface, and an edge therebetween. The case may also include atleast one image sensor embedded in the housing at an orientation suchthat when the handheld communications device is seated in the case andheld in a hand of a downward looking user with the touch-sensitivedisplay opposed to a face of the downward looking user, an optical axisof the at least one image sensor extends generally in a direction of thepath of travel. The case may further include a connector in the housingconfigured to engage a port of the handheld communications device whenthe handheld communications device is seated in the case, and at leastone conduit embedded in the case extending between and electricallyconnecting the at least one image sensor with the connector to therebyenable real-time image streaming from the at least one image sensor tothe handheld communications device. In addition, case may include atleast one processor for overlaying a visual indicator on the real-timeimage stream.

In accordance with another disclosed embodiment, a method for providingaugmented reality to a user traversing a forward path of travel whilelooking downward at a touch-sensitive display of a handheldcommunications device is provided. The method may include: acquiring animage stream from an image sensor, wherein the image sensor is fixedlyembedded in a casing of the handheld communications device at anorientation such that when the handheld communications device is held ina hand of a user with the touch-sensitive display opposed to a face ofthe downward looking user, an optical axis of the image sensor extendsgenerally in a direction of the path of travel; identifying in theacquired image stream an object in the path of travel of the user;looking-up information about the identified object; presenting, on thetouch-sensitive display in real-time the acquired image stream includingthe identified object; and presenting on the touch-sensitive display avisual indicator associated with the identified object.

In accordance with another disclosed embodiment, an image capture casingfor a handheld mobile device having an upward facing display surface andan opposing lower surface is provided. The image capture casing mayinclude a housing for surrounding at least a portion of the handheldmobile device. The image capture casing may also include at least oneimage sensor mounted in the housing and having a first optical axisextending in a first direction and second optical axis extending in asecond direction that differs from the first direction, wherein thefirst optical axis is configured to be oriented at a fixed obtuse anglewith respect to the lower surface when the handheld mobile device isretained by the housing. The image capture casing may further includecircuitry for conveying from the at least one image sensor, image datafor real-time display on the display surface.

In accordance with another disclosed embodiment, a handheld mobilecommunication device is provided. The handheld mobile communicationdevice may include a display, and a housing for retaining the displayalong an upper surface thereof and having a lower surface opposite theupper surface, and wherein an edge is located between the upper surfaceand the lower surface. The handheld mobile communication device may alsoinclude at least one first image sensor embedded in the upper surfaceand having a first optical axis facing in a first direction away fromthe upper surface, at least one second image sensor embedded in thelower surface and having a second optical axis extending in a seconddirection away from the lower surface, and at least one third imagesensor at least partially embedded in the edge and having a thirdoptical axis extending in a third direction away from the edge, andwherein the first optical axis, the second optical axis, and the thirdoptical axis diverge from each other. The handheld mobile communicationdevice may further include at least one processor within the housing forreceiving image data from the at least one first image sensor, the atleast one second image sensor and the at least one third image sensor,and for enabling presentation, on the display, of the image data fromthe at least one first image sensor, the at least one second imagesensor, and the at least one third image sensor.

In accordance with another disclosed embodiment, an image capture casingfor a mobile device having an upper display surface and an opposinglower surface is provided. The image capture casing may include ahousing for surrounding at least a portion of the mobile device, thehousing having an upper side, a lower side, and an edge interconnectingthe upper side and the lower side. The image capture casing may alsoinclude a plurality of image sensors embedded in the edge, wherein eachof the plurality of image sensors has an optical axis, and wherein whenthe mobile device is retained by the housing, each optical axis divergesfrom axes normal to the upper display surface and the lower surface. Theimage capture casing may further include circuitry for conveying imagedata from the plurality of image sensors to the display.

In accordance with another disclosed embodiment, a computer-readablemedium configured for use in a mobile communications device is provided.The computer-readable medium may contain instructions that when executedby a processor cause the processor to perform steps including: receivingfrom at least one sensor in the mobile communications device a firstindication that the mobile communications device is in forward motion;receiving from the at least one sensor in the mobile communicationsdevice a second indication that an image sensor in the mobile device isoriented for image capture; and in response to both the first indicationand the second indication, automatically causing a real-timepresentation of a forward path image to be presented on a display of themobile device.

In accordance with another disclosed embodiment, a computer-readablemedium configured for use in a mobile communications device is provided.The computer-readable medium may contain instructions that when executedby a processor cause the processor to perform steps including: receivingfrom at least one image sensor associated with the mobile communicationsdevice image data representing an environment of a user as the usermoves along a travel path; selecting a field of view window from theimage data for real-time display on the mobile communications device,wherein the field of view window corresponds to the travel path of theuser; and presenting on the display simultaneously with an interactiveuser application window, the field of view window, and omitting frompresentation on the display image data outside the field of view window.

In accordance with another disclosed embodiment, a computer-readablemedium configured for use in a mobile device is provided. Thecomputer-readable medium may contain instructions that when executed bya processor cause the processor to perform steps including: receivingfrom at least one image sensor associated with the mobile device imagedata representing an environment of a user as the user moves along atravel path; receiving from at least one sensor information reflectiveof an activity in which the user is involved; accessing in memoryindicators of a plurality of activities and an associated field of viewwindow associated with each activity; selecting a field of view windowfor real-time display on the mobile communications device, wherein theselected field of view window associated with the activity in which theuser is currently involved; and applying the selected field of viewwindow to the display of the mobile device.

In accordance with another disclosed embodiment, a computer-readablemedium configured for use in a mobile communications device is provided.The computer-readable medium may contain instructions that when executedby a processor cause the processor to perform steps including: receivingfrom at least one image sensor associated with the mobile communicationsdevice a real-time image stream from an environment of a user as theuser moves along a travel path; presenting on a touch screen display ofthe mobile communications device the real-time image stream; presentingon the touch screen display of the mobile communications devicesimultaneously with the real-time image stream, an interactive userapplication; enabling the user using the touch screen display to touch aregion associated with the real-time image stream, and cause at leastone frame of the real-time image stream to be transferred to theinteractive user application; and executing an action in the interactiveuser application that involves the at least one frame.

In accordance with another disclosed embodiment, a handheldcommunications device for displaying bi-directional video streamscaptured along transverse axes is provided. The handheld communicationsdevice may include a housing having a upper surface, a lower surface,and a side edge. The handheld communications device may also include atouch-sensitive screen located in the upper surface and at least oneprocessor programmed. The handheld communications device may alsoinclude a first image sensor having a first optical axis extending in afirst direction away from the upper surface in a manner configured tocapture a face of a user when the handheld communications device is heldin a hand of the user viewing the touch-sensitive screen, and secondimage sensor at least partially embedded in the side edge and having asecond optical axis extending in a second direction away from the sideedge in a manner configured to capture a travel path of the user whenhandheld communications device is held in the hand of the user viewingthe touch-sensitive screen, and wherein the first direction istransverse to the second direction. The at least one processor may beprogrammed to: acquire a first real-time image stream from the firstimage sensor of the face of the user; acquire a second real-time imagestream from the second image sensor of the travel path of the user; andpresent in real-time on the touch-sensitive display, a bidirectionalpresentation of the face of the user and the travel path of the user.

In accordance with another disclosed embodiment, a handheld mobilecommunications device configured to provide directional guidance to auser looking downward while traversing a forward path of travel isprovided. The handheld mobile communications device may include ahousing having a upper surface, a lower surface, and a side edge. Thehandheld mobile communications device may also include a touch-sensitivedisplay located in the upper surface, a GPS chip, and at least oneprocessor. The handheld mobile communications device may further includean image sensor fixedly embedded in the housing at an orientation suchthat when the handheld mobile device is held in a hand of a user withthe touch-sensitive display opposed to a face of the downward lookinguser, an optical axis of the image sensor extends generally in adirection of the path of travel. The at least one processor may beprogrammed to: receive from a directional guidance application a seriesof turn-by-turn directions to a destination; display on thetouch-sensitive display as the user traverses the forward path with thetouch-sensitive display opposed to the user's face, a real-time imagestream from the image sensor; and use location information obtained fromthe GPS chip, to display at least one visual indicator of theturn-by-turn directions on the touch-sensitive display simultaneouslywith the display of the real-time image stream.

In accordance with another disclosed embodiment, a removable case for ahandheld mobile phone is provided. The case may include a housing and atleast one image sensor embedded in the housing. The case may alsoinclude a connector in the housing configured to engage a port of thehandheld mobile device when the handheld mobile device is seated in thecase. The case may further include at least one conduit embedded in thecase extending between and electrically connecting the at least oneimage sensor with the connector to thereby enable real-time imagestreaming from the at least one image sensor to the handheld mobiledevice when the handheld mobile device is seated in the case.

In accordance with another disclosed embodiment, a system forcrowd-sourced generation of a street view map using image dataaggregated from a plurality of image sensors embedded in edges of mobilephones is provided. The system may include at least one processorconfigured to: maintain a street view map of a geographical area,wherein GPS coordinates are associated with locations on the street viewmap; receive from the plurality of image sensors embedded in edges of aplurality mobile phones of a plurality of users, street view images;receive from each of the plurality of mobile phones a GPS locationassociated with each of the received street view images; and update thestreet view map with received street view images from the plurality ofimage sensors when GPS locations associated with received street viewimages correspond to GPS locations on the street view map.

In accordance with another disclosed embodiment, a method forcrowd-sourced generation of a street view map using image dataaggregated from a plurality of image sensors embedded in edges of mobilephones is provided. The method may include maintaining a street view mapof a geographical area, wherein GPS coordinates are associated withlocations on the street view map; receiving from the plurality of imagesensors embedded in edges of a plurality mobile phones of a plurality ofusers, street view images; receiving from each of the plurality ofmobile phones a GPS location associated with each of the received streetview images; and updating the street view map with received street viewimages from the plurality of image sensors when GPS locations associatedwith received street view images correspond to GPS locations on thestreet view map.

In accordance with another disclosed embodiment, a computer-readablemedium configured for use in a mobile communications device is provided.The computer-readable medium may contain instructions that when executedby a processor cause the processor to perform steps including: receivinga first real-time image stream from a first image sensor having a firstoptical axis extending in a first direction; receiving a secondreal-time image stream from a second image sensor having a secondoptical axis extending in a second direction that differs from the firstdirection; determining a field of view window from the first real-timeimage stream for display on the mobile communications device; presentingon a display of the mobile communications device a real-timepresentation that corresponds with the determined field of view window;and making an adjustment to the field of view window in response to auser's interaction with the display, wherein the adjustment includes atleast one of: selecting a new field of view window from the firstreal-time image stream, and selecting a new field of view window fromthe second real-time image stream.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processor and perform any of the methodsdescribed herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1A is a schematic illustration of a user walking while staring at amobile communications device from a first view point;

FIG. 1B is a schematic illustration of the user shown in FIG. 1A from asecond view point;

FIG. 2A is a schematic illustration of a first example device consistentwith the present disclosure from a first viewpoint;

FIG. 2B is a schematic illustration of the first example device shown inFIG. 2A from a second viewpoint;

FIG. 3A is a schematic illustration of a second example deviceconsistent with the present disclosure from a first viewpoint;

FIG. 3B is a schematic illustration of the second example device shownin FIG. 3A from a second viewpoint;

FIG. 4A is a schematic illustration of the field of view of an edgecamera assembly associated with the first example device shown in FIG.2A;

FIG. 4B is a schematic illustration of the field of view of an edgecamera assembly associated with the second example device shown in FIG.3A;

FIG. 5 is a schematic illustration of the fields of view of an edgecamera assembly with a plurality of image sensors associated with thefirst example device shown in FIG. 2A;

FIG. 6 is a schematic illustration that depicts the angle of the opticalaxis of an edge camera assembly according to embodiments of thedisclosure;

FIG. 7 is a schematic illustration that depicts the orientation of theoptical axis of an edge camera assembly relative to other optical axesof cameras integral with a mobile communications device;

FIG. 8 is a schematic illustration that depicts the orientation of theoptical axis of an edge camera assembly relative to the principal axesof the mobile device of the second example device shown in FIG. 3A;

FIG. 9 is a schematic illustration that depicts the field of view of anedge camera assembly relative to other fields of view of camerasintegral to the second example device shown in FIG. 3A;

FIG. 10 is a functional block diagram illustrating the circuitry of anedge camera assembly associated with the second example device shown inFIG. 3A;

FIG. 11 is a functional block diagram illustrating the circuitry of anedge camera assembly associated with the first example device shown inFIG. 2A;

FIGS. 12A, 12B, 12C, and 12D are flowcharts of exemplary methods forproviding augmented reality to a user according to disclosedembodiments;

FIGS. 13A, 13B, 13C, 13D, 13E, and 13F are schematic illustrations ofexample usages of an edge camera assembly according to disclosedembodiments;

FIG. 14 is a schematic illustration of an example usage of an edgecamera assembly according to disclosed embodiments;

FIG. 15 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments;

FIG. 16 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments;

FIG. 17 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments;

FIG. 18 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments;

FIG. 19 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments;

FIG. 20 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments;

FIG. 21 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments; and,

FIG. 22 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts. Thedisclosure is not limited to the described embodiments and examples.Instead, the proper scope is defined by the appended claims.

Disclosed embodiments may involve an edge camera assembly configured tocapture images of the environment in front of a user for causing areal-time image presentation on a display of a mobile communicationsdevice. As used herein the term “mobile communications device” refers toany portable device with a display that can communicate with a wirelessnetwork, including, but not limited to a wireless communication chipset,a smartphone, a tablet, a smartwatch, a user equipment (UE), a personaldigital assistant, and more. It is noted that the terms “handheld mobilecommunications device,” “handheld mobile device,” “mobile communicationsdevice,” and “mobile device” may be used in the interest of brevity andmay refer to any of the variety of devices listed above.

The disclosed edge camera assembly may include a system with imagecapturing capabilities that may engage with or be part of the mobilecommunications device. The edge camera assembly may include a housing,an image sensor, and circuitry for conveying image data from the imagesensor. Consistent with disclosed embodiments, the image sensor may bedisposed in an edge surface interconnecting an upper side and a lowerside of a casing associated with the mobile communications device. In afirst aspect of the disclosure, a removable case for mobilecommunications devices is provided. In this aspect, the edge cameraassembly may be separable from the mobile communications device, butconfigured to engage with said mobile communications device when themobile communications device is seated in the case. In a second aspectof the disclosure, a mobile communications device is provided. In thisaspect, the edge camera assembly may be integral with a chassis of saidmobile communications device.

The two aspects of the present disclosure may involve an image capturecasing (also known as “casing”). The term “casing” is used herein todesignate any protective element for shielding components of the edgecamera assembly. The casing may be constituted from a resilient material(such as thermoplastic rubber, synthetic plastic, polyurethane, metal,and more) that is molded into a desired shape. In the first approach,the image capture casing may be a removable case that providesprotection to both the edge camera assembly and the mobilecommunications device. In the second approach the image capture casingmay be integrated into the mobile communications device and serves as acover of the device.

An image capture casing in accordance with embodiments of the disclosuremay include a housing for surrounding at least a portion of the mobilecommunications device. The term “housing” broadly includes any structureor enclosure designed to accommodate at least a portion of the mobilecommunications device or components of the mobile communications device.In one example, the housing may have an elongated portion that iscapable of being grasped in the hand of a user, with a lengthsubstantially longer than a width and/or thickness. The housing may havea cross-section that is square, rectangular, rounded rectangular, or anyother regular or irregular shape capable of being grasped. Such astructure may permit repeatable, consistent grasping during use. As usedin this specification and the appended claims, the terms “upper,” “top,”“lower,” “bottom,” “front,” “back,” and “rear” refer to portions of orpositions of the housing when the elongated portion of the housing isoriented horizontally and the display side of the mobile communicationsdevice faces up.

In some embodiments, the edge camera assembly may include an imagesensor mounted or embedded in the housing. The terms “mounted in thehousing” and “embedded in the housing” mean that the image sensor isbeing fully or partially encapsulated in the housing. An image sensormay be a device capable of detecting and converting optical signals inthe near-infrared, infrared, visible, and ultraviolet spectrums intoelectrical signals. The electrical signals may be used to generate imagedata. The term “image data” includes any form of data retrieved fromoptical signals in the near-infrared, infrared, visible, and ultravioletspectrums (e.g., information, image, and/or video stream). Examples ofimage sensors associated with the edge camera assembly may includedigital cameras, phone cameras, semiconductor charge-coupled devices(CCD), active pixel sensors in complementary metal-oxide-semiconductor(CMOS), or N-type metal-oxide-semiconductor (NMOS, Live MOS). Imagesensors associated with the edge camera assembly may have a field ofview greater than 90°, greater than 120°, or greater than 180°, or about360°. In addition, image sensors associated with the edge cameraassembly may have resolutions of 640×480, 1024×768, 1280×960, or anyother suitable resolution.

Consistent with the present disclosure, the image sensor may be mounted(or embedded) in the housing such that when the housing is held by awalking user the optical axis of the image sensor may generally extendin the walking direction of the user. As used herein, the term “opticalaxis” designates an imaginary line that defines a path along which lightpropagates from the center of the camera's field of view to the centerof the image sensor. In the present disclosure, the term “generallyextends (or faces) in the walking direction of the user” broadly meansthat the field of view of the edge camera assembly covers a sector ofthe user's environment that corresponds with the walking direction ofthe user. In one example, the edge camera assembly may be associatedwith a 240-degree fish-eye lens that has a wide field of view. Inanother example, the field of view of the edge camera assembly isillustrated in FIG. 1B. In both examples, since the image sensorcaptures a sector of the environment that corresponds with the walkingdirection of the user, it means that the optical axis of the imagesensor generally extends in the walking direction of the user.

The edge camera assembly may include circuitry for conveying capturedimage data from the image sensor. Consistent with disclosed embodiments,the image data may be displayed in real-time display on the mobilecommunications device. As used herein, the term “real-time” meansgenerally with no observable latency between capturing an image anddisplaying the captured image. The term “circuitry” as used hereinrefers to all of the following: (a) hardware-only circuitimplementations, such as implementations in only analog and/or digitalcircuitry (e.g., wires, electric connections); (b) combinations ofcircuits, software and/or firmware (e.g., a combination of processors,digital signal processors, software, and memories) that work together tocause an apparatus to perform various functions; and (c) circuits thatrequire software or firmware for operation (e.g., a microprocessor or aportion of a microprocessor), even if the software or firmware is notphysically present. In the first approach, the circuitry may include aconnector configured to engage a port of the mobile communicationsdevice when the mobile communications device is seated in the removablecase. In the second approach the circuitry may include a processor thatis part of the mobile communications device.

Consistent with disclosed embodiments, the edge camera assembly mayinclude or communicate with at least one processor configured to receiveimage data from the image sensor and to cause a presentation of aforward path image to be presented on a display of the mobilecommunications device. The at least one processor may constitute anyphysical device having an electric circuit that performs a logicoperation on input or inputs. For example, the at least one processormay include one or more integrated circuits, microchips,microcontrollers, microprocessors, all or part of a central processingunit (CPU), graphics processing unit (GPU), digital signal processor(DSP), field-programmable gate array (FPGA), or other circuits suitablefor executing instructions or performing logic operations. Theinstructions executed by at least one processor may, for example, bepre-loaded into a memory integrated with or embedded into the controlleror may be stored in a separate memory. The memory may comprise a RandomAccess Memory (RAM), a Read-Only Memory (ROM), a hard disk, an opticaldisk, a magnetic medium, a flash memory, other permanent, fixed, orvolatile memory, or any other mechanism capable of storing instructions.In one embodiment, the memory is configured to store indicators of aplurality of activities and an associated field of view windowassociated with each activity. This embodiment is disclosed in greaterdetail below with reference to FIG. 20.

In some embodiments, the at least one processor may include more thanone processor. Each processor may have a similar construction or theprocessors may be of differing constructions that are electricallyconnected or disconnected from each other. For example, the processorsmay be separate circuits or integrated in a single circuit. When morethan one processor is used, the processors may be configured to operateindependently or collaboratively. The processors may be coupledelectrically, magnetically, optically, acoustically, mechanically, or byother means that permit them to interact. In disclosed embodiments, theat least one processor may cause a real-time presentation of a forwardpath image on the display of the mobile communications device.Consistent with the present disclosure, the term “a real-timepresentation” may encompass a small delay from the time the image sensorcaptures the images until the images are displayed on the mobilecommunications device. For example, the delay may be smaller than about0.1 sec, smaller than about 0.05 sec, or smaller than about 0.01 sec.

FIG. 1A illustrates a situation in which a user 100 walks while staringat a mobile device 102 including a case 104. According to embodiments ofthe present disclosure, either mobile device 102 or a case 104 mayinclude an edge camera assembly. The edge camera assembly may captureimage data representing an environment of user 100 as user 100 movesalong a travel path in a direction 106. The effective visual field ofuser 100 is represented by field of view 108 and an example field ofview of the edge camera assembly is represented by field of view 110.Using the edge camera assembly, user 100 may avoid hitting an object 112located along its travel path. In one embodiment, the edge cameraassembly may cause an image presentation which shows the user theapproaching obstacle on the device screen. In another embodiment, theedge camera assembly may trigger an alert or notification responsive toan obstacle detection algorithm running on processing circuits integralor otherwise functionally associated with the edge camera assembly.

FIG. 1B illustrates the same situation illustrated in FIG. 1A from atop-view perspective. As discussed above, when mobile device 102 isseated in case 104 the optical axis of the edge camera assemblygenerally faces in walking direction 106. FIG. 1B shows that horizontalcomponent of field of view 110 covers a sector of the user's environmentthat corresponds with walking direction 106. The angular range of thehorizontal component of field of view 110 is represented in FIG. 1B forillustrative purposes by angle α. In some embodiments, the angular rangeof the horizontal component of field of view 110 may be, for example,higher than 40°, higher than 65°, higher than 90°, higher than 120°, orhigher than 180°. Alternatively, the angular range of the horizontalcomponent of field of view 110 may be, for example, less than 160°, lessthan 110°, less than 70°, or less than 45°. Consistent with the presentdisclosure, mobile device 102 may display a real-time presentation 114of the forward path that includes an indication that object 112 islocated along the user's travel path. In other embodiments, as describedin greater detail below, mobile device 102 may display overlay contenton real-time presentation 114 of the forward path.

FIGS. 2A and 2B illustrate the first aspect of the present disclosure.Specifically, FIGS. 2A and 2B illustrate an edge camera assemblyincorporated with removable case 104 that is selectively connectable andseparable from mobile device 102. The edge camera assembly includes ahousing 200 having an upper surface 202, an opposing lower surface 204,and an edge surface 206 located between upper surface 202 and lowersurface 204. Edge surface 206 may extend upward from lower surface 204.The term “edge surface” (also known as “edge” or “peripheral edge”) isused herein to denote a peripheral area interconnecting upper surface202 and lower surface 204. In some embodiments, edge surface 206 may beflat and at right angles with respect to the main surfaces of housing200. Alternatively, the edge surface may be at a rather different anglewith respect to upper surface 202 and lower surface 204 and may besomewhat curved. FIGS. 3A and 3B depict another embodiment in which theedge surface is formed other than illustrated in FIGS. 2A and 2B.Consistent with the present disclosure, the edge camera assembly mayalso include circuitry 208 and image sensor 210. For illustrationpurposes only, circuitry 208 and image sensor 210 are depictedseparately from housing 200. In fact, according to some embodiments,image sensor 210 may be fixedly secured or embedded within edge surface206, such that its optical axis is fixedly tilted at an angle other thanperpendicular to upper surface 202 and lower surface 204. For example,image sensor 210 may be fixedly located in a protrusion extending fromlower surface 204.

The edge camera assembly may include an opening (e.g., aperture 212)configured to enable an optical axis of an additional image sensor ofmobile device 102 to pass therethrough. The additional image sensor ofmobile device 102 may be distinct from image sensor 210 mounted inhousing 200. In some embodiments, removable case 104 may include abattery within housing 200 (not shown) configured to power image sensor210 and/or to provide power to mobile device 102. Accordingly, removablecase 104 may further include a receiver for wirelessly charging thebattery. Alternatively, circuitry 208 may be configured to convey powerfrom a battery within mobile device 102 to image sensor 210. In otherembodiments, edge camera assembly may also include a directionalmicrophone (not shown) substantially aimed in a direction of an opticalaxis associated with image sensor 210.

In accordance with the first aspect of the present disclosure, the edgecamera assembly may also include a connector 214 and at least oneconduit 216 that are part of or associated with circuitry 208. In someembodiments, connector 214 may be configured to engage a port of mobiledevice 102 when mobile device 102 is seated in case 104. Moreover,conduit 216 may extend between and electrically connect image sensor 210with connector 214 to enable real-time image streaming from image sensor210 to mobile device 102. In one configuration of removable case 104,image sensor 210 may be located in a portion of edge surface 206opposite connector 214, such that conduit 216 may traverse housing 200between image sensor 210 and connector 214. Consistent with the presentdisclosure, circuitry 208 may also include interface circuits (notshown) for interfacing and transferring data to mobile device 102. Thetransferred data may include image, video data, or 3D scan data acquiredby image sensor 210. In some embodiments, circuitry 208 is configured toconvey the image data over a wired connection. For example, theinterface circuits may comply with a wired standard such as USB,Micro-USB, HDMI, Micro-HDMI, Firewire, Apple, etc. In other embodiments,circuitry 208 is configured to convey the image data over a wirelessconnection. For example, the interface circuits may comply with awireless standard such as Bluetooth, WiFi, NFC, ZigBee, etc.

FIG. 2B is another illustration of the first aspect of the presentdisclosure, from a second viewpoint. The viewpoint shown in FIG. 2B isfrom a side orientation of removable case 104. In this illustration,mobile device 102 is seated in case 104; therefore the edge cameraassembly enables real-time image streaming from image sensor 210 tomobile device 102. In some embodiments, circuitry 208 may enable imagesensor 210 to be controlled by mobile device 102. For example, imagesensor 210 may be activated in response to acceleration detected by asensor in mobile device 102.

FIGS. 3A and 3B illustrate the second aspect of the present disclosure.Specifically, FIGS. 3A and 3B illustrate an edge camera assemblyintegrated into mobile device 102. Consistent with the second aspect ofthe present disclosure the edge camera assembly also includes housing200 (with upper surface 202, lower surface 204, and edge surface 206),circuitry 208 (not shown), and image sensor 210. As depicted in FIGS. 3Aand 3B, edge surface 206 may be curved and at angle other thanperpendicular to upper surface 202 and lower surface 204. In someembodiments, mobile device 102 may include a touch-sensitive display 300for presenting the real-time image stream captured by image sensor 210.Additionally, touch-sensitive display 300 together with circuitry 208can, for example, detect input from user 100. Mobile device 102 may alsoinclude a port 302, which may be an interface between a power source andcircuitry 208. In one embodiment, port 302 may be used to charge thebattery of mobile device 102. In another embodiment, port 302 may beused to connect mobile device with another electric device. For example,when the edge camera assembly is incorporated in removable case 104,port 302 may be used to form a communication channel between imagesensor 210 and mobile device 102.

Consistent with the present disclosure, edge camera assembly enablesmobile device 102 to receive from image sensor 210 a real-time imagestream from the environment of user 100, and to present ontouch-sensitive display 300 a real-time image streaming window 304simultaneously with an interactive user application window 306. Becausethere is only limited space on touch-sensitive display 300 to presentimages alongside other apps, the display real estate used for navigationmust be efficiently used. In one embodiment, mobile device 102 mayinitially present on about 50% of touch-sensitive display 300 theinteractive user application window 306 and on about 50% oftouch-sensitive display 300 real-time image streaming window 304. Inanother embodiment, mobile device 102 may initially present on at leastabout 60% of touch-sensitive display 300 the interactive userapplication window 306 and on no more than about 40% touch-sensitivedisplay 300 real-time image streaming window 304. In another embodiment,mobile device 102 may initially present on at least about 70% oftouch-sensitive display 300 the interactive user application window 306and on no more than about 30% touch-sensitive display 300 real-timeimage streaming window 304. User 100 may provide input (e.g., using GUIfeature 308) to change the ratio between interactive user applicationwindow 306 and real-time image streaming window 304. For example, user100 may change the ratio between interactive user application window 306and real-time image streaming window 304 such that real-time imagestreaming window 304 covers about 100% of of touch-sensitive display300. In another embodiment, mobile device 102 may enable user 100 toselect the field of view from the image data to be presented inreal-time image streaming window 304. Mobile device 102 may initiallydisplay a field of view that is estimated to correspond with the travelpath of user 100 (e.g., using image processing algorithms that identifyobjects in the image data). But in some cases, user 100 may desire tochange the field of view presented on real-time image streaming window304. This embodiment is discussed in greater detail below with referenceto FIG. 15.

Several types of mobile device 102 may further include a firstadditional image sensor 310 having an optical axis directed transverseto upper surface 202 (i.e., the selfie camera) and a second additionalimage sensor 312 embedded beneath lower surface 204 having an opticalaxis directed transverse to lower surface 204 (i.e., the back camera).Consistent with the present disclosure, mobile device 102 may receive,via circuitry 208, image data from image sensor 210, first additionalimage sensor 310, and second additional image sensor 312. In oneembodiment, user 100 may switch between the image sensors, such thatreal-time image streaming window 304 may present image data from imagesensor 210, first additional image sensor 310, or second additionalimage sensor 312. This embodiment is discussed in greater detail belowwith reference to FIG. 16. In another embodiment, mobile device 102 maycause a simultaneous image presentation on touch-sensitive display 300from at least two image sensors out of: image sensor 210, firstadditional image sensor 310, and second additional image sensor 312.Simultaneous image presentation may result from aggregating image datacaptured by image sensor 210 and image data captured by secondadditional image sensor 312 such that an aggregated image may bedisplayed on touch-sensitive display 300. Alternatively, simultaneousimage presentation may include streaming images in each window from adifferent image sensor. This embodiment enabled mobile device 102 topresent in real-time on touch-sensitive display 300, a bidirectionalpresentation of the face of user 100 and the environment in front ofuser 100. The bidirectional presentation may be shared or posted online.This embodiment is discussed in greater details below with reference toFIG. 17.

FIGS. 4A and FIG. 4B are perspective illustrations of edge cameraassemblies according to the first and second aspects of the disclosure.Both figures illustrate an example field of view 110 of image sensor 210when mobile device 102 is held in a hand 400 of user 100 withtouch-sensitive display 300 opposed to a face 402 of the downwardlooking user 100. As shown, field of view 110 may extend generally in adirection of the path of travel. In one example, field of view 110 maycapture a sector of between 20 to 80 degrees relative to a vertical axisof the user's standing body. According to both aspects of thedisclosure, the direction of field of view 110 may be adjustable. Forexample, field of view 110 may be adjusted by physically changing theorientation of mobile device 102 or by physically changing a relativedirection of an aperture of image sensor 210 relative to mobile device102. In some embodiments, the physical orientation of image sensor 210may also be changed manually or by using electromechanical actuators.Alternatively, the field of view 110 may be adjusted digitally bychanging filtration of the output from one or more image sensors 210.

FIG. 5 is a perspective illustration of an edge camera assemblyaccording to the first aspect of the disclosure. In this illustration,the edge camera assembly includes a plurality of image sensors 210embedded in edge surface 206. Each image sensor 210 (e.g., 210A and210B) may be associated with optics (such as lens, filter, and more) andoptionally an illuminator. Embodiments including a structured lightilluminator may include a corresponding 3D decoder, which may providedepth information for points on objects imaged by the edge cameraassembly. In some embodiments, each of the plurality of image sensors210 has an associated field of view (e.g., field of view 110A and fieldof view 110B) and the field of view of at least one of the plurality ofimage sensors 210 overlaps with the field of view of another of theplurality of image sensors 210. In addition, the edge camera assemblymay be associated with a processor for receiving image information fromat least two of the overlapping image sensors 210, and for unifyingimage information from the at least two overlapping image sensors 210.For example, the processor may stitch together the image informationfrom the at least two overlapping image sensors 210. Mobile device 102may also present on touch-sensitive display 300 the unified imageinformation from the at least two overlapping image sensors 210. Inother embodiments, mobile device 102 may construct a virtualthree-dimensional image using image data acquired from the at least twooverlapping image sensors 210.

Using a plurality of image sensors 210 enable capturing a greater fieldof view 110 and obtaining more information. For example, thestereoscopic arrangement illustrated in FIG. 5 may provide depthinformation for points on the objects (e.g., object 112) captured by theedge camera assembly. Although only two image sensors 210 are shown, oneskilled in the art would recognize that more than two image sensors 210may be used, such as three image sensors 210, four image sensors 210,and more. Accordingly, the combined field of view of the plurality ofimage sensors 210 may be greater than 180°, e.g., about 240°, about300°, or about 360°. There may be a large number of possiblearrangements of the plurality of image sensors 210 in housing 200. Forexample, in a first configuration, edge camera assembly may include twoimage sensors 210: a first in a region of a right top corner, and asecond in a region of a left top corner. In a second configuration, edgecamera assembly may include three image sensors 210: a first in a topedge, a second in a region of a right top corner, and a third in aregion of a left top corner. In a third configuration, edge cameraassembly may include a first image sensor 210 with an optical viewfacing at least partially to an outward side of a top edge portion, asecond image sensor 210 with an optical view facing at least partiallyoutward of a right side edge portion, and a third image sensor 210 withan optical view facing at least partially outward of a left side edgeportion.

FIG. 6 is a schematic illustration that depicts the angle of the opticalaxis of an edge camera assembly. Consistent with the disclosure above,the edge camera assembly may be incorporated in mobile device 102 or inremovable case 104. In one embodiment, image sensor 210 mounted inhousing 200 may have at least one optical axis 600 configured to beoriented at a fixed obtuse angle with respect to lower surface 204. Anobtuse angle is one which is more than 90° but less than 180°. In FIG.6, the obtuse angle of optical axis 600 is represented by angle β. Asdiscussed above, image sensor 210 may be fixedly secured or embeddedwithin edge surface 206, therefore optical axis 600 may also be fixed.In other words, user 100 cannot change the physical orientation ofoptical axis 600 without damaging mobile device 102. In one embodiment,housing 200 may include at least one channel therein (not shown) forretaining image sensor 210, and wherein an axis of the channel extendsfrom housing 200 at an obtuse angle with respect to a portion of lowersurface 204 when mobile device 102 is retained by housing 200. In someexamples, angle β may be about 150° (i.e., any value between 145° to155°), about 160 (i.e., any value between 155 to 165), or about 170(i.e., any value between 165 to 175).

Typically, when housing 200 is held by a walking user 100 it is at anacute angle with respect to a plane 602 on which user 100 walks. Anacute angle is one which is more than 0° but less than 90°. In FIG. 6,the holding angle of housing 200 is represented by angle γ. Consistentwith the present disclosure, image sensor 210 may be oriented such thatwhen housing 200 is held at holding angle γ at least one optical axis600 generally faces in walking direction 106. In contrast to angle β,angle γ is not fixed. When user 100 walks, the holding angle of housing200 may constantly change. However, typically when mobile device 102 isheld by a walking user 100 the holding angle γ remains between 0° and90°. For example, holding angle γ may be about 25° (i.e., any valuebetween 20° to 30°), about 35° (i.e., any value between 30° to 40°), orabout 45° (i.e., any value between 40° to 50°). Due to the changingnature of the holding angle γ, the direction of optical axis 600 cannotbe parallel to the exact walking direction 106. But, as described above,as long as the horizontal component of field of view 110 covers apredefined portion (e.g., a sector) of the user's environment thatcorresponds with walking direction 106, at least one optical axis 600 isconsidered to generally face walking direction 106.

FIG. 7 is a schematic illustration that depicts the orientation ofoptical axis 600 relative to other optical axes of cameras integral tomobile device 102. Consistent with present disclosure, mobile device 102may include first additional image sensor 310 having a first additionaloptical axis 700 facing in a first direction away from upper surface202, and second additional image sensor 312 having a second additionaloptical axis 702 extending in a second direction away from lower surface204. First additional image sensor 310 may be embedded in a side ofhousing 200 opposite to the side of second additional image sensor 312.Typically, first additional optical axis 700 and second additionaloptical axis 702 are substantially opposite to each other, such that theangle between them is about 180°. Consistent with embodiments of thedisclosure, optical axis 600 of image sensor 210, which is at leastpartially embedded in edge surface 206, extends in at least one thirddirection away from edge surface 206. As illustrated in FIG. 7, opticalaxis 600, first additional optical axis 700, and second additionaloptical axis 702 may diverge from each other. Specifically, optical axis600 and first additional optical axis 700 may be oriented at an obtuseangle with respect to each other. The angle between optical axis 600 andfirst additional optical axis 700 is represented by angle δ. In someexamples, angle δ may be about 110° (i.e., any value between 100° to120°). Similarly, optical axis 600 and second additional optical axis702 may be oriented at an acute angle with respect to each other. Theangle between optical axis 600 and second additional optical axis 702 isrepresented by angle ε. In some examples, angle ε may be about 40°(i.e., any value between 30° to 50°).

FIG. 8 illustrates the orientation of optical axis 600 relative to theprincipal axes of mobile device 102. Herein, the term “principal axes”refers to a set of three mutually perpendicular axes in mobile device102 about which the moment of inertia is at a maximum. For the purposeof simplicity and clarity, the disclosure assumes that mobile device 102has substantially homogeneous mass distribution and that the central ofmass converts with the center of symmetry 800. The principal axes mayinclude a horizontal axis (i.e., X-axis), a longitudinal axis (i.e.,Y-axis), and a lateral axis (i.e., Z-axis). The set of principal axes ofmobile device 102 also defines three central axial planes, namely X-Yplane, X-Z plane, and Y-Z plane.

As shown in FIG. 8, mobile device 102 may include first image sensor210A and second image sensor 210B, each of which may be fixedly embeddedor secured in peripheral edge surface 206. In contrast to firstadditional image sensor 310 (and to second additional image sensor 312),first image sensor 210A and second image sensor 210B may have a fixedoptical axis at a non-orthogonal angle relative to each of a first sideof housing 200 (e.g., upper surface 202), a second side of the housing(e.g., lower surface 204), and a central axial plane (e.g., X-Y plane,X-Z plane, and Y-Z plane). Specially, first image sensor 210A isassociated with optical axis 600A that is at a non-orthogonal angle ζ₁relative to central axial plane X-Y, and second image sensor 210B isassociated with optical axis 600B that is at a non-orthogonal angle ζ₂relative to central axial plane Y-Z. In one example, optical axis 600Amay be tilted at an angle ζ₁ between about 5° and about 80° relative tothe central axial plane X-Y. In another example, optical axis 600B maybe tilted at an angle ζ₂ between about 10° and about 70° relative to thecentral axial plane Y-Z. In one embodiment, the horizontal and/orvertical component of field of view 110 of each of image sensors 210Aand 210B may be greater than 60°, greater than 75°, greater than 90°, orgreater than 105°. Accordingly, the field of view of first image sensor210A may partially overlap with a field of view of second image sensor210B.

FIG. 9 is a schematic illustration that depicts field of view 110relative to other fields of view of cameras integral to mobile device102. Consistent with the present disclosure, mobile device 102 mayinclude at least one image sensor 210 (or be associated with at leastone image sensor 210 when using removable case 104). At least one imagesensor 210 may have at least one optical axis 600 extending in a firstdirection from the peripheral edge surface 206. Mobile device 102 mayinclude first additional image sensor 310 embedded in housing 200 andhaving optical axis 700 extending in a second direction from uppersurface 202 and second additional image sensor 312 embedded in housing200 having optical axis 702 extending in a third direction from lowersurface 204. As depicted, first additional image sensor 310 may beassociated with a field of view 900 and second additional image sensor312 may be associated with a field of view 902. As mentioned above withreference to FIG. 6, optical axis 600 may be oriented at a fixed obtuseangle with respect to lower surface 204. Additionally, as mentionedabove with reference to FIG. 7, optical axis 600 may be oriented at afixed obtuse angle with respect to first additional optical axis 700 andat a fixed acute angle with respect to second additional optical axis702.

In one embodiment, field of view 110 partially overlaps with field ofview 902 of second additional image sensor 312. In the removable caseimplementation, field of view 110 partially overlaps with field of view902 of second additional image sensor 312 when mobile device 102 isseated in case 104. When image sensor 210 is integral with mobile device102, field of view 110 may constantly partially overlap with field ofview 902 of second additional image sensor 312. Mobile device 102 mayinclude at least one processor within housing 200 configured to stitchtogether image data from at least one image sensor 210 and secondadditional image sensor 312. Specifically, the at least one processormay receive image data from at least one image sensor 210 and secondadditional image sensor 312, and apply stitching algorithms that takeinto account the relative position of the image sensors 210 and 312 andmap the displacement of pixels in the different images. The stitchingalgorithm may be used to combine two (or more) images. For example, thestitching algorithm may include identifying an overlap area between twoimages, detecting and matching key points in the image data, selecting aprojection surface, and more. Alternatively, at least one processor maybe configured to receive image data from each of the image sensors(i.e., 210, 310, and 312) to enable presentation on touch-sensitivedisplay 300 of at least a portion of the image data received from atleast one image sensor 210, first additional image sensor 310, andsecond additional image sensor 312. In one example, the at least oneprocessor may simultaneously receive image data from at least two of theat least one image sensor 210, first additional image sensor 310, andsecond additional image sensor 312, and cause a simultaneous imagepresentation on touch-sensitive display 300 from the at least two ofsaid image sensors. Example of the simultaneous image presentation ispresented in FIG. 17.

FIG. 10 is a functional block diagram of an edge camera assemblyassociated with the second example device shown in FIG. 3A. Consistentwith the present disclosure, mobile device 102 may include a processingdevice 1005, a network interface 1015, a memory interface 1002, and aperipherals interface 1004. These components can be separated or can beintegrated in one or more integrated circuits. The various components inmobile device 102 can be coupled by one or more communication buses orsignal lines.

Sensors, devices, and subsystems can be coupled to the peripheralsinterface 1004 to facilitate multiple functionalities. For example,image sensor 210, first additional image sensor 310, second additionalimage sensor 312, and a positioning system (e.g., GPS receiver) 1014 canbe coupled to the peripherals interface 1004. Other sensors 1016, suchas a depth sensor, a motion sensor, a light sensor, a proximity sensor,a temperature sensor, and a biometric sensor, can also be connected toperipherals interface 1004 to facilitate related functionalities.

Communication functions may be facilitated through network interface1015, which may be an Ethernet port connected to radio frequencyreceivers and transmitters and/or optical (e.g., infrared) receivers andtransmitters. The specific design and implementation of networkinterface 1015 depends on the communications network(s) over whichmobile device 102 is intended to operate. For example, in someembodiments, mobile device 102 includes network interface 1015 designedto operate over a GSM network, a GPRS network, an EDGE network, a Wi-Fior WiMax network, and a Bluetooth® network.

I/O system 1020 may include a touch screen controller 1021, audiocontroller 1022, and/or other input controller(s) 1023. Touch screencontroller 1021 may be coupled to a touch-sensitive display 300.Touch-sensitive display 300 and touch screen controller 1021 can, forexample, detect contact, movement or break thereof using any of aplurality of touch sensitivity technologies, including but not limitedto capacitive, resistive, infrared, and surface acoustic wavetechnologies as well as other proximity sensor arrays or other elementsfor determining one or more points of contact with the touch-sensitivedisplay 300. Touch-sensitive display 300 can also, for example, be usedto implement virtual or soft buttons and/or a keyboard.

Audio controller 1022 may be coupled to a speaker 1026 and a microphone1028 to facilitate voice-enabled functions such as voice recognition,voice replication, digital recording, and telephony functions. The otherinput controller(s) 1023 may be coupled to other input/control devices1030 such as one or more buttons, rocker switches, thumb-wheel, infraredport, USB port, and/or a pointer device such as a stylus.

Memory interface 1002 may be coupled to a memory device 1010. Memorydevice 1010 may include high-speed random access memory and/ornon-volatile memory such as one or more magnetic disk storage devices,one or more optical storage devices, and/or flash memory (e.g., NAND,NOR). Memory device 1010 may store an operating system 1032, such asAndroid, iOS, RTXC Quadros, Linux, WINDOWS, or an embedded operatingsystem such as VXWorkS. The operating system 1032 can includeinstructions for handling basic system services and for performinghardware dependent tasks.

Memory device 1010 may also store communication instructions 1034 tofacilitate communicating with one or more additional devices, one ormore computers and/or one or more servers. The memory device 1010 caninclude graphical user interface instructions 1036 to facilitate graphicuser interface processing; sensor processing instructions 1038 tofacilitate sensor-related processing and functions; phone instructions1040 to facilitate phone-related processes and functions; messaginginstructions 1042 to facilitate electronic-messaging related processesand functions; web browsing instructions 1044 to facilitate webbrowsing-related processes and functions; media processing instructions1046 to facilitate media processing-related processes and functions;GPS/navigation instructions 1048 to facilitate GPS andnavigation-related processes and instructions; camera instructions 1050and/or other software instructions 1052 to facilitate other processesand functions.

In accordance with the present disclosure, when mobile device 102 isheld in the hand of user 100 with touch-sensitive display 300 opposed tothe face of the downward looking user 100, field of view 110 of imagesensor 210 may extend generally in a direction of the path of travel.Image sensor 210 may therefore acquire an image stream along thedirection of the path of travel. In some embodiments, sensor processinginstructions 1038 include instructions to enable processing device 1005to acquire the image stream from image sensor 210. In one embodiment,media processing instructions 1046 may include instructions to enableprocessing device 1005 to identify in the acquired image stream anobject in the path of travel of user 100. The identified object may be,for example, a person, a sales establishment, or a fire hydrant. Inaddition, consistent with one embodiment of the present disclosure,communication instructions 1034 include instructions to enableprocessing device 1005 to look-up information about the identifiedobject. For example, processing device 1005 can look up a person's namewhen the identified object is a person. For the example where theidentified object is a sales establishment, processing device 1005 canlook up details about products or services available at the salesestablishment.

In some embodiments, processing device 1005 can present ontouch-sensitive display 300, in real-time, the acquired image streamincluding the identified object. Further, processing device 1005 canpresent on touch-sensitive display 300, one or more visual indicatorsassociated with the identified object. The term “visual indicator” meansa displayed item or representation of content, for example, text, icons,graphics. For example, when the identified object is a salesestablishment, the visual indicators may include the name of the salesestablishment and an icon representing the sales products of the salesestablishment. In some embodiments, the visual indicator may representinformation about a commercial offer, for example, information about anon-going sale at the exemplary sales establishment. The visual indicatormay be overlaid on the acquired real-time image stream. For example, thevisual indicator representing an on-going sale at the exemplary salesestablishment may be overlaid on the part of the real-time image streamcorresponding to the sales establishment. In some embodiments, an offsetmay be added during overlay of the visual indicator to avoid obscuringarea of interest in the image stream. Further, the visual indicator mayinclude an indication that more information is available uponinteraction with touch-sensitive display 300. For example, a clickablelink. In response to detection of an interaction, processing device 1005can be further programmed to execute an action. For example, in responseto a touch-detection on a clickable link, processing device 1005 candisplay on touch-sensitive display 300, details of on-going sale at theexemplary sales establishment.

In some embodiments, other software instructions 1052 includeinstructions to enable processing device 1005 to determine whether user100 is on a collision course with the identified object. As an example,processing device 1005 can determine that user 100 is on a collisioncourse with a fire hydrant identified in the acquired image stream. Inresponse to a determination of a collision, processing device 1005 maygenerate a warning of the collision course. The warning may be providedto user 100 using a visual indicator on touch-sensitive display 300, anaudio warning using speaker 1026, a tactile warning using a tactileactuator included in other input/control devices 1030, or a combinationof two or more warning systems. In response to the warning, user 100 cantake evasive action to avoid collision with the fire hydrant.

In some embodiments, processing device 1005 can display on a first partof touch-sensitive display 300, the real-time image stream including theidentified object, and display on a second part of touch-sensitivedisplay 300, simultaneously with the real-time image stream, aninteractive user application. Processing device 1005 may initiallypresent on about 50% of touch-sensitive display 300 the real-time imagestream including the identified object and on about 50% oftouch-sensitive display 300 the interactive user application. In anotherembodiment, processing device 1005 may initially present on at leastabout 60% of touch-sensitive display 300 the real-time image streamincluding the identified object and on no more than 40% oftouch-sensitive display 300 the interactive user application. User 100may provide input, using a GUI feature on touch-sensitive display 300,to change the real-time image stream including the identified object andthe interactive user application. An example of the interactive userapplication is a messaging application. User 100 may use the messagingapplication to send and receive messages and to communicate with anotheruser. In some embodiments, media processing instructions 1046 includeinstructions to enable user 100 to select an image from the acquiredimage stream using processing device 1005. For example, an image ofexemplary sales established acquired in image stream from image sensor210. Further, messaging instructions 1042 may include instructions toenable user 100 to send the selected image to the other user usingprocessing device 1005 via the messaging application.

In some embodiments, GPS/navigation instructions 1048 includeinstructions to enable processing device 1005 to generate a turn-by-turndirections to a destination using location information obtained frompositioning system 1014. As user 100 traverses the forward path with thetouch-sensitive display opposed to the user's face processing device1005 can display on touch-sensitive display 300, a real-time imagestream from image sensor 210. Processing device 1005 may use locationinformation obtained from positioning system 1014 to display at leastone visual indicator associated with the turn-by-turn directions on thetouch-sensitive display simultaneously with the display of the real-timeimage stream. The turn-by-turn directions may be associated with a routeof travel to the destination. In some embodiments, the visual indicatormay include information about places along the route of travel.Specifically, processing device 1005 may provide visual indicatorsassociated with places located at no more than a first predetermineddistance from the route of travel and/or no more than a secondpredetermined distance from a current location of user 100. The secondpredetermined distance may be larger than the first predetermineddistance. For example, processing device 1005 may provide visualindicators associated with places located at no more than 300 metersfrom the route of travel and no more than 500 meters from a currentlocation of user 100. In other embodiments, positioning system 1014comprises a compass and GPS/navigation instructions 1048 further includeinstructions to enable processing device 1005 to use the directionalinformation obtained from the compass (in addition to the locationinformation) to display the visual indicator associated with theturn-by-turn directions. The visual indicator may include an arrowindicating turn-by-turn directions, a representation of a compass, anumber representing a distance to a next turn, a name of a salesestablishment, or a name of a street.

In some embodiments, GPS/navigation instructions 1048 includeinstructions to enable processing device 1005 to receive locationinformation from positioning system 1014. As user 100 traverses theforward path with the touch-sensitive display opposed to the user'sface, processing device 1005 can display on touch-sensitive display 300,a real-time image stream from image sensor 210. Further, processingdevice 1005 may use location information obtained from positioningsystem 1014 to display at least one visual indicator on touch-sensitivedisplay 300. As an example, processing device 1005 may use locationinformation obtained from positioning system 1014 to display a visualindicator associated with a restaurant located in proximity to user 100.The real-time image stream and the visual indicator may be displayed ondifferent parts of touch-sensitive display 300. In some embodiments, thevisual indicator may be overlaid on the real-time image stream. In oneembodiment, GPS/navigation instructions 1048 and other softwareinstructions 1052 may enable processing device 1005 to generate augmentreality content (e.g., visual indicators) on the real-time imagestreaming window 304 based on information derived from the interactiveuser application presented on interactive user application window 306.Specifically, the interactive user application may be a messagingapplication and the information may be derived using linguistic analysistechniques such as semantic-based text recognition, vector spaceanalysis, rule-based analysis, statistical analysis, or other knowntechniques. For example, the information may include an indication abouta status of the user 100, such that user 100 is hungry. Thereafter,processing device 1005 may present one or more visual indicatorsassociated with restaurants in the area of user 100 overlaid on thereal-time image stream.

FIG. 11 is a functional block diagram of an edge camera assemblyassociated with the first example device shown in FIG. 2A. Consistentwith the present disclosure, removable case 104 may include a sensorunit 1110. Sensor unit 1110 may include a depth sensor and image sensor210, sensor optics 1112, and sensor driving circuits 1114. Sensor unit1110 may further include an illuminator 1122, associated illuminatoroptics 1124, and illuminator driving circuits 1120. Illuminator 1122 maybe, for example, a flash light source or a structured light source.Sensor unit 1110 may also comprise a controller 1116 and an interface1118. Controller 1116 controls sensor driving circuits 1114 andilluminator driving circuits 1120. Consistent with this embodiment, theedge camera assembly may further comprise control circuits 1132,location based service circuits 1134, and image processing circuits1136. One or more of control circuits 1132, location based servicecircuits 1134, and image processing circuits 1136 may be provided aspart of the removable case or may be integrated into mobile devicecircuits 1150 of mobile device 102. Controller 1116 may communicate withcontrol circuits 1132 via interface 1118. User interfaces of mobiledevice 102 comprise touch-sensitive display 300, speaker 1026, andtactile actuators 1142. A person skilled in the art would recognize thatan edge camera assembly associated with removable case 104 may executeall the functions described above with reference to the edge cameraassembly associated with mobile device 102.

FIG. 12A depicts an exemplary method 1200 for providing augmentedreality to user 100, in accordance with example embodiments of thepresent disclosure. In one embodiment, at least some of the steps ofmethod 1200 may be performed by mobile device 102 illustrated in FIG.10. In another embodiment, at least some of the steps of method 1200 maybe performed by sensor unit 1110 illustrate in FIG. 11. In the followingdescription, reference is made to certain components of FIG. 10 and FIG.11 for purposes of illustration. It will be appreciated, however, thatother implementations are possible and that other components may beutilized to implement the exemplary method.

At step 1202, processing device 1005 (or control circuits 1132) mayacquire an image stream along the direction of the path of travel fromimage sensor 210. At step 1204, processing device 1005 (or imageprocessing circuits 1136) may identify, in the acquired image stream, anobject in the path of travel of user 100. The identified object may be,for example, a person, sales establishment, or a bus stop. At step 1206,processing device 1005 (or mobile device circuits 1150) may look-upinformation about the identified object. In one example, when theidentified object is a person, processing device 1005 can look up aperson's name in the memory device 1010. In another example, when theidentified object is a bus stop, processing device 1005 can look updetails about the next bus that is scheduled to arrive. In oneembodiment, the step of looking-up information about the identifiedobject, may include transmitting to a remote server an image thatincludes the identified object, and the remote server retrievesinformation about the identified object. At step 1208, processing device1005 (or control circuits 1132) may present on touch-sensitive display300, in real-time, the acquired image stream including the identifiedobject. At step 1210, processing device 1005 (or control circuits 1132)may present on touch-sensitive display 300, a visual indicatorassociated with the identified object. For example, when the identifiedobject is a sales establishment, the visual indicators may include thename of the sales establishment and an icon representing the salesproducts of the sales establishment. In some embodiments, the visualindicator may represent information about a commercial offer, forexample, information about an on-going sale at the exemplary salesestablishment. The visual indicator may be overlaid on the acquiredreal-time image stream. For example, the visual indicator representingan on-going sale at the exemplary sales establishment may be overlaid onthe part of the real-time image stream corresponding to the salesestablishment.

FIG. 13A-13C are schematic illustrations of an example usage of an edgecamera assembly according to disclosed embodiments. In FIG. 13A anacquired image stream 1312 is displayed on a touch-sensitive display300. A visual indicator 1314 (representing the chocolate kitchen in thisexample) may be overlaid on acquired image stream 1312. Further, thevisual indicator may include an indication that more information isavailable upon interaction with touch-sensitive display 300 (forexample, icon 1316). In response to a touch interaction corresponding toicon 1316, further details about the identified object may be displayedon touch-sensitive display 300. For example, further details 1322 aredisplayed in FIG. 13B in response to touch-interaction with icon 1316.In addition, details 1322 may include an indication that walkingguidance is available upon a further interaction with touch-sensitivedisplay 300 (for example, icon 1324). In response to a touch interactioncorresponding to icon 1324, a map 1332 (as shown in FIG. 13C) isdisplayed on interactive user application window 306. In this example,the map may include a route from user 100 location to the chocolatekitchen.

FIG. 12B depicts an exemplary method 1220 for avoiding collision of user100 with an identified object, in accordance with example embodiments ofthe present disclosure. In one embodiment, all of the steps of method1220 may be performed by mobile device 102 illustrated in FIG. 10. Inanother embodiment, all of the steps of method 1220 may be performed bysensor unit 1110 illustrate in FIG. 11. In the following description,reference is made to certain components of FIG. 10 and FIG. 11 forpurposes of illustration. It will be appreciated, however, that otherimplementations are possible and that other components may be utilizedto implement the exemplary method.

At step 1222, processing device 1005 (or control circuits 1132) mayacquire an image stream along the direction of the path of travel fromimage sensor 210. At step 1224, processing device 1005 (or imageprocessing circuits 1136) may identify, in the acquired image stream, anobject in the path of travel of user 100. The identified object may be,for example, a lamp post. At step 1226, processing device 1005 (or imageprocessing circuits 1136) may determine whether user 100 is on acollision course with the identified object. As an example, processingdevice 1005 may determine that user 100 is on a collision course withthe lamp post identified in the acquired image stream. At step 1228,processing device 1005 may generate a warning of the collision course.The warning may be provided to user 100 using a visual indicator ontouch-sensitive display 300, an audio warning using speaker 1026, atactile warning using a tactile actuator included in other input/controldevices 1030, or a combination of two or more warning systems. Inresponse to the warning, user 100 can take evasive action to avoidcollision with the lamp post. In another embodiment, image processingcircuits 1136 can determine whether user 100 is on a collision coursewith the identified object. In response to a determination of acollision, control circuits 1132 may generate a warning of the collisioncourse. The warning may be provided to user 100 using a visual indicatoron touch-sensitive display 300, an audio warning using speaker 1026, atactile warning using tactile actuators 1142, or a combination of two ormore warning systems.

FIG. 13D is a schematic illustration of an example usage of an edgecamera assembly according to disclosed embodiments. In FIG. 13D anacquired image stream 1312 is displayed on a touch-sensitive display300. In this example, visual indicator 1342 is a warning that user 100is on a collision course with the lamp post. As depicted, the visualindicator 1342 may be displayed on interactive user application window306 to make sure that user 100 notices the warning.

FIG. 12C depicts an exemplary method 1240 for providing augmentedreality to user 100, in accordance with example embodiments of thepresent disclosure. In one embodiment, all of the steps of method 1240may be performed by mobile device 102 illustrated in FIG. 10. In anotherembodiment, all of the steps of method 1240 may be performed by sensorunit 1110 illustrate in FIG. 11. In the following description, referenceis made to certain components of FIG. 10 and FIG. 11 for purposes ofillustration. It will be appreciated, however, that otherimplementations are possible and that other components may be utilizedto implement the exemplary method. At step 1242, processing device 1005(or control circuits 1132) may acquire an image stream along thedirection of the path of travel from image sensor 210. At step 1244,processing device 1005 (or control circuits 1132) may acquire locationinformation from positioning system 1014 (or location based servicecircuits 1134). As an example, the location information may includelocation information of user 100, location information associated with aroute of travel to a destination. At step 1246, processing device 1005(control circuits 1132) may present on touch-sensitive display 300, inreal-time, the acquired image stream. At step 1248, processing device1005 (or control circuits 1132) may present on touch-sensitive display300, a visual indicator simultaneously with the display of the real-timeimage stream. In one example, the visual indicator in this embodimentmay be associated with turn-by-turn directions. In another example, thevisual indicator in this embodiment may be associated with a placelocated in proximity to user 100.

FIG. 13E is a schematic illustration of an example usage of an edgecamera assembly according to disclosed embodiments. An acquired imagestream 1312 is displayed on one part of a touch-sensitive display 300.Further, a visual indicator 1352, representing turn-by-turn directionsto a destination is overlaid on top of acquired image stream 1312.

FIG. 12D depicts an exemplary method 1260 for providing augmentedreality to user 100, in accordance with example embodiments of thepresent disclosure. Method 1260 may be used, for example, in a gameproviding augmented reality by combining an acquired image stream withgame graphics. In one embodiment, all of the steps of method 1260 may beperformed by mobile device 102 illustrated in FIG. 10. In anotherembodiment, all of the steps of method 1260 may be performed sensor unit1110 illustrate in FIG. 11. In the following description, reference ismade to certain components of FIG. 10 and FIG. 11 for purposes ofillustration. It will be appreciated, however, that otherimplementations are possible and that other components may be utilizedto implement the exemplary method. At step 1262, processing device 1005(or control circuits 1132) may acquire an image stream along thedirection of the path of travel from image sensor 210. At step 1264,processing device 1005 (or control circuits 1132) may present ontouch-sensitive display 300, in real-time, the acquired image stream. Atstep 1266, processing device 1005 (or control circuits 1132) may presenton touch-sensitive display 300, a visual indicator. In an exemplarygame, the visual indicator may be a game character overlaid on top ofthe acquired image stream.

FIG. 13E is a schematic illustration of an example usage of an edgecamera assembly according to disclosed embodiments. An acquired imagestream 1312 is displayed on one part of a touch-sensitive display 300.Further, a visual indicator 1362, representing a game character isoverlaid on top of acquired image stream 1312. The game character may ormay not be associated with the application displayed on interactive userapplication window 306.

FIG. 14 is a schematic illustration of an example usage of an edgecamera assembly according to disclosed embodiments. In this example, theedge camera assembly enables “dragging and dropping” images to anongoing application. Specifically, the edge camera assembly may beassociated with a computer-readable medium configured for use in mobiledevice 102. The computer-readable medium contains instructions that whenexecuted by processing device 1005 cause processing device 1005 toperform steps including: receiving from at least one image sensor 210 areal-time image stream from an environment of user 100 as user 100 movesalong a travel path; presenting on touch-sensitive display 300 thereal-time image stream; and presenting on touch-sensitive display 300simultaneously with the real-time image stream, an interactive userapplication. For example, mobile device 102 may present real-time imagestreaming window 304 simultaneously with interactive user applicationwindow 306. The steps of this embodiment further include: enabling user100 using touch-sensitive display 300 to touch a region associated withthe real-time image stream, and cause at least one frame 1400 of thereal-time image stream to be transferred to the interactive userapplication; and executing an action in the interactive user applicationthat involves the at least one frame 1400.

In related embodiments, the interactive user application may be amessaging application and the instructions further include sending amessage to a remote recipient, wherein the message includes at least oneframe 1400. In one embodiment, at least one frame 1400 may be a singleframe captured by at least one image sensor 210, e.g., an image. Inanother embodiment, at least one frame 1400 may be multiple framescaptured, e.g., a live video stream captured by at least one imagesensor 210. In a suggested implementation of this embodiment, the touchmay include pressing GUI feature 1402 to select at least one frame 1400following by a drag and drop motion from the region associated with thereal-time stream (e.g., image streaming window 304) to a regionassociated with the interactive user application (e.g., interactive userapplication window 306). In addition, the instructions may furtherinclude buffering the real-time stream and enabling user 100 to interactwith touch-sensitive display 300 in a manner permitting a bufferedportion of the real-time image stream to be transmitted via themessaging application.

FIG. 15 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments. Since the holdingorientation of mobile device 102 can vary from person to person, therelevant portion of field of view of each user also changes. Therefore,in another embodiment of the disclosure, a method is presented forproviding an adjustable window that corrects for the angular orientationof mobile device 102 to enable images of the immediate path of user 100to be presented on touch-sensitive display 300. The adjustable windowalso enables user 100 the freedom to select the field of view to see inreal-time image streaming window 304. For example, using pinch gestureson real-time image streaming window 304 or using a specific GUI feature,user 100 can select the image data to be presented. The selection of theimage data may be considered as cropping the received image data.Specifically, the edge camera assembly may be associated with acomputer-readable medium configured for use in mobile device 102. Thecomputer-readable medium contains instructions that when executed byprocessing device 1005 cause processing device 1005 to perform stepsincluding receiving from at least one image sensor 210 image datarepresenting an environment of user 100 as user 100 moves along a travelpath 1500. Image 1502 is an example of the image data received fromimage sensor 210.

The steps may further include selecting a field of view window 1504 fromthe image data for real-time display on mobile device 102. The selectionof the field of view window 1504 may be executed manually by user 100 bydetecting user input from a control permitting the user to select thefield of view window. Alternatively, the selection may include detectingan angular orientation of mobile device 102 and automatically adjustingfield of view window 1504 based on the angular orientation. In addition,the selection may include detecting a movement pattern of mobile device102 as user 100 moves along travel path 1500 and automatically adjustingfield of view window 1504 to compensate for movements of mobile device102. In one example, selected field of view window 1504 may correspondto travel path 1500. The steps may further include presenting ontouch-sensitive display 300 simultaneously with interactive userapplication window 306, selected field of view window 1504, and omittingfrom presentation on touch-sensitive display 300 image data outside thefield of view window. Consistent with present disclosure, theinstructions may further include adjusting field of view window 1504 inreal-time.

FIG. 16 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments. In this example, theedge camera assembly enables mobile device 102 to switch between three(or more) camera inputs using a simple gesture. Specifically, the edgecamera assembly may include one or more image sensors 210 and may beassociated with a computer-readable medium configured for use in mobiledevice 102. The computer-readable medium contains instructions that whenexecuted by processing device 1005 cause processing device 1005 toperform steps including: receiving a first real-time image stream 1600from a first image sensor (e.g., image sensor 210) having a firstoptical axis (e.g., optical axis 600) extending in a first direction;receiving a second real-time image stream 1602 from a second imagesensor (e.g., first additional image sensor 310) having a second opticalaxis (e.g., optical axis 700) extending in a second direction thatdiffers from the first direction; and receiving a third real-time imagestream 1604 from a third image sensor (e.g., second additional imagesensor 312) having a third optical axis (e.g., optical axis 702)extending in a third direction that differs from the first and seconddirections. In the discussion above, the method includes receiving imagedata from all three image sensors 210 associated with mobile device 102;however, one skilled in the art would recognize that this embodimentwill work with less cameras.

The steps further include: determining a field of view window from firstreal-time image stream 1602 for display on mobile device 102; presentingon touch-sensitive display 300 a real-time presentation that correspondswith the determined field of view window (e.g., selected field of viewwindow 1504); and making an adjustment to field of view window 1504 inresponse to a user's interaction with touch-sensitive display 300,wherein the adjustment includes at least one of: selecting a new fieldof view window 1504 from first real-time image stream 1600; selecting anew field of view window 1504 from the second real-time image stream1602; and selecting a new field of view window 1504 from the thirdreal-time image stream 1604. In one embodiment, the adjustment to fieldof view window 1504 includes a continuous motion on the display. Thecontinuous motion may be along two distinct directions, wherein a firstmovement along a first direction changes the field of view window of areal-time image stream and a second movement along a second directionchanges the source of real-time image stream.

FIG. 17 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments. In this example, theedge camera assembly enables mobile device 102 to present, store, and/orshare bidirectional image data that is captured along transverse opticalaxes. For example, a video of the environment in front of user 100 maybe captured simultaneously with images of the user's face. Specifically,the edge camera assembly may be associated with mobile device 102 fordisplaying bidirectional video streams captured along transverse axes.Mobile device 102 may include housing 200 having an upper surface 202, alower surface 204, and a side edge (e.g., edge surface 206). Mobiledevice 102 may further include touch-sensitive display 300 located inupper surface 202, a first image sensor (e.g., first additional imagesensor 310) having a first optical axis extending in a first directionaway from upper surface 202 in a manner configured to capture a face ofuser 100 when the mobile device 102 is held in the hand of user 100viewing touch-sensitive display 300. The mobile device may also includea second image sensor (e.g., image sensor 210) at least partiallyembedded in the side edge and having a second optical axis extending ina second direction away from the side edge in a manner configured tocapture a travel path 1500 of user 100 when mobile device 102 is held inthe hand of user 100 viewing touch-sensitive display 300, and whereinthe first direction is transverse to the second direction. Mobile device102 may also include at least one processor (e.g., processing device1005 programmed to: acquire a first real-time image stream from thefirst image sensor of the face of user 100; acquire a second real-timeimage stream from the second image sensor of travel path 1500; andpresent in real-time on touch-sensitive display 300, a bidirectionalpresentation of the face of user 100 and travel path 1500.

In related embodiments, the bidirectional presentation may include afirst window 1700 for the first image stream and a second window 1702for the second image stream. Further to the discussion above regardingreal-time image streaming window 304, at least one of first window 1700and second window 1702 may be resizable via touch interaction with thetouch-sensitive display 300. For example, second window 1702 may beabout 15%, 30%, or 50% of touch-sensitive display 300. In addition, aposition of at least one of first window 1700 and the second window 1702is movable via touch interaction with the touch-sensitive display. Inother related embodiments, the at least one processor may store thebidirectional presentation for a defined period of time, wirelesslycommunicate a real-time bidirectional presentation to a remoterecipient, wirelessly upload a real-time bidirectional presentation to aserver, and/or cause the bidirectional presentation to be transferred toan interactive user application. In addition, the at least one processormay transfer the bidirectional presentation upon identifying a drag anddrop motion from a region associated with bidirectional presentation toa region associated with the interactive user application.

FIG. 18 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments. In this example, amethod is provided for self-activating the edge camera assembly. Forexample, when processing device 1005 receives a signal that mobiledevice 102 is moving forward in an appropriate orientation (such as at atilt angle indicative of holding), processing device 1005 mayautomatically activate the edge camera assembly and cause image data toappear on touch-sensitive display 300. Specifically, the edge cameraassembly may be associated with a computer-readable medium configuredfor use in mobile device 102. The computer-readable medium containsinstructions that when executed by processing device 1005 causeprocessing device 1005 to perform the following steps. Step 1800includes receiving from at least one sensor in mobile device 102 a firstindication that mobile device 102 is in forward motion. In oneembodiment, the at least one sensor may include an accelerometer, andthe first indication includes a measure relating to acceleration. Forexample, the first indication is a velocity of between about 1 km/h and35 km/h identified for more than a predetermined period of time. Step1804 includes receiving from the at least one sensor in mobile device102 a second indication that an image sensor in mobile device 102 isoriented for image capture (e.g., image sensor 210). In one embodiment,the at least one sensor includes a sensor that detects tilt wherein thesecond indication corresponds to a tilt angle of mobile device 102. Forexample, the tilt angle may be between about 0° and about 45° relativeto a surface on which a user of the mobile communications device walks.Alternatively, the at least one sensor may include a proximity sensor,wherein the second indication is reflective of whether at least oneimage sensor 210 is unobstructed. In step 1806 and in response to boththe first indication and the second indication, the method includesautomatically causing a real-time presentation of a forward path imageto be presented on touch-sensitive display 300.

In related embodiments, the instructions are configured to causeautomatic capture when a velocity of mobile device 102 is detected toexceed a predetermined threshold, when a tilt angle of mobile device 102is within a predetermined range, when a velocity of mobile device 102 isdetected to fall beneath a predetermined threshold, and/or when avelocity of mobile device 102 is detected to exceed a predeterminedthreshold. In addition, the instructions are configured to allow user100 to select predetermined thresholds for image capture, thepredetermined thresholds including at least one of velocity and theorientation of mobile device 102. In one embodiment, the instructionsfurther include detecting whether touch-sensitive display 300 is active,and automatically causing the real-time presentation of the forward pathimage to be presented on touch-sensitive display 300 in response to acombination of a detection of an active display and in response to boththe first indication and the second indication. In another embodiment,the instructions further include determining whether a battery status ofmobile device 102 is higher than a predefined threshold, andautomatically causing the real-time presentation of the forward pathimage to be presented on touch-sensitive display 300 in response to acombination of a determination that the battery status is higher thanthe predefined threshold and in response to both the first indicationand the second indication.

FIG. 19 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments. In this example, theedge camera assembly is incorporated in the casing of mobile device 102,such that one image sensor may be used to capture image data from twodirections. As illustrated in FIG. 19, the multi-directional camera maybe embedded in the bezel of a mobile device 102. Specifically, in thisembodiment an image capture casing is provided. The image capture casingmay include an upward facing display surface (e.g., touch-sensitivedisplay 300) and an opposing lower surface (e.g., lower surface 204).The image capture casing may include housing 200 for surrounding atleast a portion of mobile device 102, image sensor 210, and circuitryfor conveying from image sensor 210, image data for real-time display onthe display surface. In this embodiment, image sensor 210 may be mountedin housing 200 and have a first optical axis extending in a firstdirection and second optical axis extending in a second direction thatdiffers from the first direction, wherein the first optical axis isconfigured to be oriented at a fixed obtuse angle with respect to thelower surface when mobile device 102 is retained by housing 200.Consistent with this embodiment, image sensor 210 may be associated withan optical apparatus 1900 embedded in housing 200 to enable image sensor210 to capture image data in a first direction extending from theperipheral edge and in a second direction extending from the upwardfacing display surface. In a first configuration the angle between thefirst optical axis and the second optical axis may be obtuse. In asecond configuration the angle between the first optical axis and thesecond optical axis may be acute. Examples of the angles are discussedabove with reference to FIGS. 6-9.

FIG. 20 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments. In this example, theedge camera assembly enables mobile device 102 to change field of viewwindow 1504 based on an activity of user 100. For example, a mobiledevice mounted on a bicycle might require a field of view window that isdifferent from a field of view of a user who is walking. In thisembodiment, a method is provided for detecting the activity of the userand altering the windowed field of view based on that detected activity.Specifically, the edge camera assembly may be associated with acomputer-readable medium configured for use in mobile device 102. Thecomputer-readable medium contains instructions that when executed byprocessing device 1005 cause processing device 1005 to perform stepsincluding: receiving from image sensor 210 image data representing anenvironment of user 100 as user 100 moves along travel path 1500; andreceiving from at least one sensor information reflective of an activityin which user 100 is involved. The information reflective of theactivity may include an indication of velocity, an indication ofacceleration, and/or an indication of tilt angle of mobile device 102.The plurality of activities may include at least two of: walking 2000,bike riding 2002, car driving 2004, motorcycle riding 2006, and ridingany other motored vehicle, such as, segway, scooter, hoverboard, andmore. The steps of this embodiment further include: accessing in memoryindicators of a plurality of activities and an associated field of viewwindow is associated with each activity; selecting field of view window1504 for real-time display on mobile device 102, wherein the selectedfield of view window is associated with the activity in which user 100is currently involved; and applying the selected field of view window tothe display of the mobile communications device. In one embodiment,selecting field of view window 1504 may involve executing imagerecognition algorithms to exclude from field of view window 1504 imagedata beyond boundaries associated with a path of user 100.

FIG. 21 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments. Presently, onlinemap services (e.g., Google maps street view) provide images that arelong out of date. This is because companies that maintain online mapsare able to capture street view images once every few months or years.However, a community of users constantly using mobile devices 102 withedge camera assemblies may constantly capture images of theirsurroundings. This information may be used to update online map serviceswith street views much more regularly and/or to crowd-source a 3Dscanning index of the environment of the community of users. In oneexample, a system for crowd-sourced generation of a street view mapand/or a 3D scanning index using image data aggregated from a pluralityof image sensors 210 embedded in edges of mobile device 102 is provided.The system comprises a processing device 1005 configured to maintain astreet view map and/or a 3D scanning index of a geographical area,wherein GPS coordinates are associated with locations on the street viewmap and/or the 3D scanning index. Processing device 1005 is furtherconfigured to: receive from plurality of image sensors 210 embedded inedges of a plurality mobile devices 102 of a plurality of users 100,image data (e.g., street view images that may be received in real-timeor at a later time); and receive from each of plurality of mobiledevices 102 a GPS location associated with the received image data.Thereafter, processing device 1005 may update the street view map and/orthe 3D scanning index with received image data from the plurality ofimage sensors when GPS locations associated with received street viewimages correspond to GPS locations on the street view map and/or the 3Dscanning index.

In related embodiments, processing device 1005 may receive the streetview images from the plurality of mobile phones after the mobile devices102 have altered the street view images to obscure personal information.For example, processing device 1005 may receive the street view imagesfrom plurality of mobile devices 102 without metadata informationassociated with plurality of users 100. Processing device 1005 may alsorequest images of specific locations of interest and/or area ofinterest. In addition, processing device 1005 may receive a time stampassociated with each of the street view image and when multipleoverlapping street view images are received from multiple users 100, toupdate the map with the latest street view image. Processing device 1005may also unify multiple overlapping street view images that are receivedfrom multiple users 100, and update the map with a unified street viewimage. The following example is depicted in FIG. 21, illustrating oneimplementation of this embodiment. In this example, server 2100 maydetermine that the current street view image of shop 2102 in thedatabase is not updated. Then server 2100 sends an indication to mobiledevice 102 requesting an updated image of shop 2102. Mobile device 102,using image sensor 210, may capture an image of shop 2102 without theuser intervention and in one alternative even without the userknowledge. The street view image may be transferred to server 2100 vianetwork 2104 and thereafter uploaded to the online map service 2108.

FIG. 22 is a schematic illustration of another example usage of an edgecamera assembly according to disclosed embodiments. Consistent with thepresent the disclosure, a computer-implemented method is presented forproviding a convenient way for user 100 to access interactive userapplications that may interact with image data captured by image sensor210. In one example, after user 100 presses GUI feature 2200 located inreal-time image streaming window 304 a plurality of icons (e.g., icon2202) associated with multiple interactive user applications may bepresented in real-time image streaming window 304. User 100 may have theoption to select which icons to display in real-time image streamingwindow 304. In one embodiment, after user 100 presses one of the iconsassociated with the multiple interactive user applications, thecorresponding interactive user application may be opened in interactiveuser application window 306. One example of an interactive userapplication that interact with image data captured by image sensor 210is described above with reference to FIG. 14.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer-readable media,such as secondary storage devices, for example, hard disks or CD ROM, orother forms of RAM or ROM, USB media, DVD, Blu-ray, or other opticaldrive media.

Computer programs based on the written description and disclosed methodsarc within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

1-180. (canceled)
 181. A computer-readable medium configured for use ina mobile communications device, the computer-readable medium containinginstructions that when executed by a processor cause the processor toperform steps, comprising: receiving from at least one sensor in themobile communications device a first indication that the mobilecommunications device is in forward motion; receiving from the at leastone sensor in the mobile communications device a second indication thatan image sensor in the mobile device is oriented for image capture; andin response to both the first indication and the second indication,automatically causing a real-time presentation of a forward path imageto be presented on a display of the mobile device.
 182. Thecomputer-readable medium of claim 181, wherein the instructions furthercomprising, during the real-time presentation of the forward path image,displaying additional non-image information on the display.
 183. Thecomputer-readable medium of claim 181, wherein the at least one sensorincludes an accelerometer, and the first indication includes a measurerelating to acceleration.
 184. The computer-readable medium of claim181, wherein the first indication is a velocity of between about 2 km/hand 35 km/h measured for a predetermined period of time.
 185. Thecomputer-readable medium of claim 181, wherein the at least one sensorincludes a sensor that detects tilt, and wherein the second indicationcorresponds to a tilt angle of the mobile device.
 186. Thecomputer-readable medium of claim 181, wherein the tilt angle is betweenabout 0° and about 45° relative to a surface on which a user of themobile communications device walks.
 187. The computer-readable medium ofclaim 181, wherein the at least one sensor includes a proximity sensor,and the wherein the second indication is reflective of whether the atleast one image sensor is unobstructed.
 188. The computer-readablemedium of claim 181, wherein the instructions are configured to causeautomatic capture when a velocity of the mobile device is detected toexceed a predetermined threshold.
 189. The computer-readable medium ofclaim 181, wherein the instructions are configured to cause automaticcapture when a tilt angle of the mobile device is within a predeterminedrange.
 190. The computer-readable medium of claim 181, wherein thewherein the instructions are configured to cease automatic image capturewhen a velocity of the device is detected to fall beneath apredetermined threshold.
 191. The computer-readable medium of claim 181,wherein the instructions are configured to cease automatic image capturewhen a velocity of the device is detected to exceed a predeterminedthreshold.
 192. The computer-readable medium of claim 181, wherein theinstructions are configured to allow a user to select predeterminedthresholds for image capture, the predetermined thresholds including atleast one of velocity and mobile device orientation.
 193. Thecomputer-readable medium of claim 181, wherein the instructions furtherinclude detecting whether the display is active, and automaticallycausing the real-time presentation of the forward path image to bepresented on the display of the mobile device in response to acombination of a detection of an active display and in response to boththe first indication and the second indication.
 194. Thecomputer-readable medium of claim 181, wherein the instructions furtherinclude determining whether a battery status of the mobilecommunications device is higher than a predefined threshold, andautomatically causing the real-time presentation of the forward pathimage to be presented on the display of the mobile device in response toa combination of a determination that the battery status is higher thanthe predefined threshold and in response to both the first indicationand the second indication.
 195. The computer-readable medium of claim181, wherein the real-time presentation includes a video streamcomprising multiple forward path images.